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1.  Introduction 


Expanding  the  use  of  commercial  items^  in  Department  of  Defense  (DoD)  systems  offers  the 
DoD  opportunities  for  reduced  cycle  time,  faster  insertion  of  new  technology,  lower  life  cycle 
costs,  greater  reliability  and  availability,  and  support  from  a  more  robust  industrial  base.  It  is  a 
fact  that  for  many  of  the  technologies  that  are  critical  to  military  systems,  the  commercial 
marketplace — and  not  the  DoD — ^now  drives  the  pace  of  innovation  and  development.  The 
increasing  priority  on  the  use  of  commercial  items^  in  DoD  systems  is  reflected  in  DoD 
Directive  5000.1,  which  states  that  the  use  of  commercial  items  in  DoD  systems  is  the 
preferred  approach  for  meeting  operational  requirements.  Simply  put,  if  the  DoD  intends  to 
field  state-of-the-art  systems  in  a  cost-effective  manner,  then  it  must  incorporate  commercial 
items  into  these  systems. 

Major  DoD  programs  have  been  successful  in  building  systems  that  realized  the  benefits  of 
using  commercial  items.  These  programs  found  that  the  use  of  commercial  items  required  a 
reemphasis  of  some  traditional  DoD  business,  management,  and  engineering  practices,  as  well 
as  a  number  of  changes  to  other  practices.  Successful  programs  embraced  these  changes  by 
building  systems  that  were  conceived,  acquired,  and  sustained  with  an  understanding  of  the 
imperatives  of  the  commercial  marketplace. 

This  document  provides  guidance  on  the  use  of  commercial  items  for  program  managers  as 
well  as  for  integrated  product  teams  and  contractors  that  support  the  program  manager.  It 
provides  an  overview  of  the  fundamental  challenges  that  organizations  face  when  they 
integrate  commercial  items  to  form  a  system.  It  then  addresses  the  issues  involved  in  buying 
from  the  commercial  marketplace,  summarizes  lessons  learned  from  programs  that  have  made 
extensive  use  of  commercial  items,  and  offers  suggestions. 

It  is  left  to  each  program  manager  to  determine  how  to  implement  the  suggestions  eontained 
in  this  document  in  a  manner  consistent  with  appropriate  laws  and  regulations.  The 
information  and  suggestions  provided  here  are  derived  from  real  program  experiences  in 
which  managers  implemented  commercially  based  solutions.  These  experiences  reflect  a  wide 
range  of  program  types:  programs  where  the  buyer  was  a  commercial  entity  (e.g.,  a  bank); 
programs  in  which  the  government  directly  acquired  commercial  items;  and  programs  where  a 
contractor  was  responsible  for  acquiring  and  integrating  commercial  items.  Most  of  the 
programs  studied  were  software  intensive.  However,  a  number  of  hardware  programs  were 
also  considered.  Many  of  the  lessons  are  not  unique  to  situations  in  which  programs 
incorporate  commercial  items.  Rather,  they  address  concepts  that  should  be  considered  during 
any  system  acquisition.  These  lessons  have  been  included  because  programs  that  implemented 
commercially  based  solutions  identified  them  as  significant. 


'  Commercial  items  are  defined  in  the  Federal  Acquisition  Regulation  (FAR),  Part  2.  The  language  has  been 
paraphrased  in  the  definitions  below  and  is  repeated  in  full  in  the  appendix. 

^  Commercial  items  are  a  subset  of  the  non-developmental  item  (NDI)  category.  Many  of  the  fundamentals  and 
lessons  learned  presented  in  this  document  also  apply  to  the  broader  definition  of  NDIs.  However,  the  NDIs  not 
included  in  the  FAR  definition  of  “commercial  item”  (such  as  items  developed  for  another  program  that  are 
reused)  are  outside  the  scope  of  this  document. 
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Terms  and  Definitions 

There  is  a  lack  of  precision  in  the  definitions  used  for  the  term  “commercial  items.”  As  such, 
the  following  section  clarifies  some  of  the  major  terms  used  in  this  document. 

Business  practices  are  the  tasks,  duties,  and  functions  performed  to  support  the  objectives 
of  an  organization. 

A  commercial  item  is  one  customarily  used  for  nongovernmental  purposes  that  has  been 
or  will  be  sold,  leased,  or  licensed  (or  offered  for  sale,  lease,  or  license)  to  the  general 
public.  An  item  that  includes  modifications  customarily  available  in  the  commercial 
marketplace  or  minor  modifications'^  made  to  meet  federal  government  requirements  is 
still  a  commercial  item.  In  addition,  services  such  as  installation,  maintenance,  repair,  and 
training  that  are  procured  for  support  of  an  item  described  above  are  considered 
commercial  items  if  they  are  offered  to  the  public  under  similar  terms  and  conditions  or 
sold  competitively  in  substantial  quantities  based  on  established  eatalog  or  market  prices. 

A  commercial  off-the-shelf  (COTS)  item  is  one  that  is  sold,  leased,  or  licensed  to  the 
general  public;  offered  by  a  vendor  trying  to  profit  from  it^;  supported  and  evolved  by  the 
vendor^  who  retains  the  intellectual  property  rights;  available  in  multiple,  identical  copies; 
and  used  without  modification  of  the  internals.’ 

A  contractor  is  a  company  or  institution  that  is  under  contract  to  the  government  and  from 
whom  a  program  manager  expects  to  receive  a  delivered  system  as  specified  in  a  contract. 
A  contractor  may  also  be  a  vendor. 

n 

End  users  are  those  people  who  will  be  using  the  system  in  the  operational  environment. 

The  marketplace  is  the  aggregation  of  buyers  and  sellers  where  goods  are  offered  for  sale. 

A  stakeholder  is  any  person  or  organization  that  is  affected  by  or  has  an  impact  on  a 
system  or  decision.’ 

System  context  encompasses  all  those  considerations  that  define  and  constrain  the  system 
to  be  fielded:  functional  and  non- functional  requirements,  end-user  practices,  business 
drivers,  operational  environment,  constraints,  applicable  policies,  budgets,  and  schedules.’ 

A  vendor  is  a  commercial  enterprise  whose  purpose  in  producing  a  product  is  to  offer  it 
for  sale  in  the  marketplace,  and  not  in  response  to  specific  program  needs.  The  vendor 
may  also  be  a  contractor  or  subcontractor  who  is  under  eontract  to  modify  a  commercial 
item  in  response  to  unique  program  requirements. 


^  This  definition  is  paraphrased  from  the  FAR,  Part  2.  The  complete  definition  is  included  as  Appendix  A. 

Minor  modifications  are  defined  in  the  definition  of  commercial  items  in  FAR,  Part  2.  See  Appendix  A. 

’  This  distinguishes  the  item  from  components  that  are  built  by  a  commercial  entity  for  its  own  use  and  are 
subsequently  offered  to  the  program,  but  not  to  the  wider  commercial  marketplace. 

*  The  item  could  also  be  supported  under  special  license  agreement  such  that  the  vendor  retains  responsibility  for 
the  product. 

’  This  is  defined  in  COTS-Based  Systems  for  Program  Managers.  See  [CPM  99], 
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2.  Using  Commercial  Items;  Some  Fundamentals _ 

Using  commercial  items  in  DoD  systems  is  not  new.  Most  systems  being  developed  today  use 
some  commercial  items  (e.g.,  computer  hardware,  operating  systems,  database  management 
systems,  and  even  batteries,  engines,  and  air  eonditioners).  What  is  new  is  the  wider 
availability  of  commercial  items  and  the  desire  and  need  to  increase  the  use  of  these  items  in 
DoD  systems  in  order  to  provide  the  DoD,  particularly  the  war  fighter,  with  the  latest 
available  technology. 

The  extent  to  which  individual  programs  will  use  commercial  items  will  vary.  Some  programs 
will  integrate  a  few  commercial  items  within  a  largely  custom-built  DoD  system.  Other 
programs  will  find  a  commercial  item  from  a  single  vendor  that  can  largely  replace  a  custom 
DoD  system.  Still  other  programs  will  build  systems  that  are  integrated  fi’om  multiple 
commercial  items  purchased  from  different  vendors. 

There  is  no  single  set  of  rules  that  covers  this  broad  range  of  possibilities.  Deciding  how 
commercial  items  affect  a  specific  program  depends  on  the  degree  to  which  the  program 
intends  to  use  commercial  items,  the  extent  to  which  introducing  the  commercial  item  alters 
the  physical  characteristics  of  the  system^,  and  the  complexity  of  integrating  commercial  and 
custom  DoD  items.  There  may  be  several  competing  approaches,  and  the  program  manager 
must  determine  which  is  most  appropriate.  Regardless  of  the  approach  selected,  some 
common  fundamentals  have  been  observed  in  programs  that  have  used  commercial  items. 


First,  increased  reliance  on  commercial  items  implies  a  different  paradigm  of  system 
acquisition.  The  most  fundamental  ehange  involves  the  dynamic  interaction  between  the 
system  context,  the  system  architecture  and  design,  and  the  commercial  items  available  in  the 
marketplace.  Managing  this  interaction  requires  unprecedented  cooperation  among  the 
program  offiee,  the  stakeholders,  the  contractor,  and  in  many  cases  the  vendor  in  order  to 
effect  the  tradeoffs  necessary  to  keep  the  program  on  track.  The  changes  from  the  traditional 
acquisition  paradigm  are  illustrated  below. 

Traditional  Model  Recommended  Model 


®  For  example,  flat  panel  displays  have  been  successfully  introduced  in  airplane  cockpits.  Alternately,  a  new 
engine  on  an  airplane  may  impact  aircraft  stability  or  other  airplane  components. 
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The  cooperation  depicted  in  the  recommended  model  is  fundamental  to  implementing  many 
elements  of  acquisition  reform — not  just  those  involving  commercial  items.  All  programs 
benefit  from  close  working  relationships  among  the  various  parties.  Unfortunately,  many 
programs  (including  those  making  use  of  commercial  items)  continue  to  follow  a  model  akin 
to  the  traditional  model  where  an  attempt  is  made  to  fully  specify  requirements  before  design 
alternatives  and  marketplace  exigencies  are  considered.  If  a  program  is  to  maximize  its 
opportunities  to  benefit  from  the  commercial  market,  then  marketplace  technologies,  products 
and  dynamics  must  influence  many  aspects  of  the  system  context  (including  requirements), 
the  architecture  and  design,  and  the  acquisition  strategy.  In  short,  the  goal  in  design  of  a 
commercial-based  system  must  be  to  adapt  requirements  to  the  capabilities  available  in  the 
marketplace  rather  than  adapting  commercial  capabilities  to  DoD  requirements. 

Second,  the  marketplace,  not  the  program  manager,  drives  development  of  the  commercial 
item.  Development  of  commercial  items  is  driven  primarily  by  the  vendors’  perceptions  of 
what  will  sell  to  the  largest  number  of  potential  users.  This  ean  be  an  advantage  beeause  a 
DoD  program  does  not  have  to  directly  fund  performance  or  functional  enhancements  to 
eommereial  items.  Vendors  commonly  implement  such  enhancements  in  order  to  retain  or 
inerease  market  share.  However,  reliance  on  the  marketplace  also  means  that  program 
managers  must  reeognize  that  a  vendor  may  remove  capabilities  that  are  important  to  a  DoD 
program,  and  other  capabilities  that  are  not  needed  by  the  program  may  be  added.  While  the 
program  manager  does  not  directly  control  the  characteristics  of  a  commercial  item,  as  occurs 
when  a  contractor  or  subcontractor  develops  a  custom  product,  this  does  not  imply  that  the 
program  manager  has  no  control  over  the  system  to  be  deployed. 

The  program  manager  must  conform  to  the  behavior  of  the  other  buyers  in  the  marketplace, 
and  then  exert  control  by  managing  and  verifying  requirements  in  a  manner  that  optimizes  the 
use  of  eommereial  items — often  by  adopting  the  requirements  of  the  other  buyers  as  elosely 
as  is  practical.  Market  research  must  be  perfonned  to  evaluate  the  capabilities  of  available 
commercial  items,  the  performance  of  vendors,  and  the  relative  size  of  the  program  to  the 
vendor’s  business  base.  Business  relationships  must  be  established  with  eontraetors  and 
vendors  to  ensure  that  program  needs  are  communicated  in  a  manner  that  maximizes  the 
program’s  leverage.  Finally,  the  system  must  be  engineered  to  accommodate  marketplace- 
driven  changes  to  commercial  items  throughout  the  system  life  eyele. 

Third,  the  difference  between  integrating  commercial  items  and  developing  a  custom 
capability  is  fundamental.  In  custom  development,  the  program  directs  the  behavior  of 
system  components  and  the  interfaces  among  components.  Program  managers  who  use 
commercial  items  have  little  insight  into  how  the  commercial  items  are  put  together,  how  they 
behave,  and  why.  Commercial  items  are  built  around  vendor-specific  assumptions,  such  as 
unique  approaehes  to  error  handling,  data  access,  and  interaction,  in  the  case  of  software.  The 
details  of  these  assumptions  are  typically  unavailable  to  the  program  manager  and  are  likely 
to  differ  from  those  of  other  system  components.  Identifying  the  assumptions  of  commercial 
items  and  developing  a  strategy  for  working  with  (and  around)  these  assumptions  makes 
integration  ehallenging. 

Finally,  using  commercial  items  means  that  many  acquisition  activities  are  repeated 
throughout  the  life  of  the  program.  Frequent  changes  driven  by  the  marketplace  are  likely  to 
make  activities  typical  of  sustainment  necessary  even  before  initial  system  delivery.  Similarly, 
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activities  typical  of  development  may  be  repeated  after  system  deployment  because  a  system 
based  on  commercial  items  is  never  really  “complete.”  In  some  sense,  system  development 
and  sustainment  activities  merge.  The  opportunity  to  enhance  system  performance  or 
capabilities  through  rapid  technology  insertion  is  one  of  the  motivators  for  using  commercial 
items.  However,  new,  changed,  and  obsolete  commercial  items  necessitate  repeated  cycles  of 
requirement  definition,  commercial  item  evaluation,  and  system  engineering.  Some  form  of 
replanning  and  reengineering  will  be  ongoing  throughout  the  life  of  the  system. 

Numerous  acquisitions  have  stumbled  for  lack  of  careful  consideration  of  the  above 
fundamentals.  However,  there  are  logical  remedies  to  the  unique  risks  imposed  by 
commercial  items — and  those  risks,  when  addressed  correctly,  can  be  far  outweighed  by  the 
benefits.  As  a  first  step  in  identifying  and  mitigating  these  risks,  the  program  manager  should 
imderstand  the  lessons  learned  by  similar  programs  and  determine  how  these  experiences  can 
enhance  program  acquisition. 
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3.  The  Implications  of  These  Fundamentals:  Some  Key  Lessons  Learned 


The  programs  that  have  incorporated  commercial  items  are  diverse.  In  spite  of  this  diversity,  a 
few  common  themes  can  be  identified  in  their  experiences.  In  the  following  sections,  common 
themes  involving  business  practices,  commercial  item  evaluation,  buying  practices,  and  life- 
cycle  engineering  are  discussed  in  detail  in  terms  of  individual  lessons  learned.  Examples 
from  actual^  programs  are  provided  to  illustrate  the  lessons.  Suggestions  for  the  program 
manager  conclude  each  section. 

3.1  Embracing  Commercial  Business  Practices 

The  use  of  commercial  items  frequently  means  embracing  commercial  business  practices  that 
are  embedded  in  the  commercial  item.  The  commercial  item  embodies  the  vendor’s 
expectation  of  how  it  will  be  used.  This  includes  the  concept  of  operation  it  supports,  interface 
and  data  standards,  architecture  and  design,  and  the  characteristics  of  form,  fit,  and  function. 
Equally  important  are  the  vendor’s  business  practices  and  strategies  in  areas  such  as 
development,  maintenance,  distribution  of  updates,  and  availability  of  spare  parts.  Many  DoD 
requirements  must  be  adjusted  to  accommodate  both  the  vendor’s  anticipated  uses  of  the 
commercial  item  and  the  vendor’s  business  practices  in  order  to  maximize  the  item’s 
effectiveness  in  meeting  program  needs. 

3.1.1  A  gap  will  exist  between  DoD  and  commercial  use — and  the  gap  may  be  large.  The 
program  manager  and  the  stakeholders  must  define  and  bridge  the  gap  between  the  DoD 
system  context  and  the  commercial  use  anticipated  by  the  vendor  through  investigation  and 
negotiation.  In  one  case,  a  successful  program  was  able  to  maximize  the  use  of  commercial 
items  through  extensive  negotiation  with  the  program’s  stakeholders.  Together  they  made 
environmental  concessions  (e.g.,  decreased  operating  temperature  and  shock  parameters)  to 
facilitate  the  use  of  commercial  items.  Another  successful  program  influenced  commercial 
buyers  to  adopt  DoD  practices  in  the  marketplace.  In  several  other  programs  the  extent  of  the 
gap  was  not  well  understood  by  any  of  the  parties  involved.  Vendors,  program  offices,  and  the 
contractors  believed  that  commercial  items  provided  most  of  the  required  capability,  when  in 
reality  the  items  provided  more  limited  capability.  One  program  made  no  serious  effort  to 
estimate  the  gap  between  the  services  provided  by  the  preferred  commercial  solution  and  the 
actual  DoD  requirements  because  program  managers  believed  that  a  commercial  solution  was 
mandated  by  high-level  DoD  policy. In  all  of  these  cases,  program  offices  and  contractors 
later  discovered  that  commercial  items  lacked  essential  capabilities.  Adding  these  capabilities 
required  expensive  custom  development,  and  resulted  in  cost  and  schedule  overruns  that  could 
have  been  avoided  had  the  programs  instituted  meaningful  and  open  communication  among 
the  vendors,  the  contractors  and  all  of  the  program  stakeholders. 

3.1.2  DoD  standards  and  compliance  documents  may  restrict  the  use  of  commercial  items. 
While  the  use  of  military  standards  has  declined  substantially,  the  need  for  interoperability 
between  weapon  and  command  and  control  systems  has  introduced  a  new  set  of  standards. 

’  To  maintain  confidentiality,  the  names  and  identifying  details  of  the  programs  are  omitted.  Programs  represent 
both  major  weapon  systems  and  information  systems  from  both  the  DoD  and  commercial  industry. 

For  a  summary  of  DoD  policies  related  to  commercial  items,  see  [POLICY]. 


6 


For  instance,  the  DoD  has  specified  requirements  among  information  systems  for  software 
infrastructure,  reusable  components,  and  interfaces  to  supporting  systems.  The  Joint  Technical 
Architecture  (JTA)  and  the  Defense  Information  Infrastructure/Common  Operating 
Environment  (DII/COE)  are  examples  of  these.  The  JTA  and  the  DII/COE  demand  that  key 
architectural  decisions*^  be  made  before  commercial  items  are  selected,  and  that  the 
commercial  items  used  must  conform  to  those  decisions.  However,  many  programs  have 
found  that  this  demand  constrains  the  choice  of  available  commercial  items.  One  program 
identified  a  similar  problem  with  DoD  data  standards,  where  only  a  small  (less  than  10%) 
match  was  found  between  the  relevant  standards  and  the  data  definitions  employed  by 
commercial  items. 

3.1.3  Modifying  the  commercial  items  is  not  the  best  way  to  bridge  the  gap.  Some  programs 
failed  because  of  a  firm  expectation  that  commercial  items  should  be  modified  to 
accommodate  program  requirements.  Like  many  DoD  programs,  one  private  corporation  fell 
into  the  trap  of  modifying  most  of  its  commercial  items  in  order  to  give  them  a  unique 
corporate  flavor.  As  a  result  of  the  practice,  many  of  the  corporate  programs  modifying 
commercial  items  experienced  recurring  technical  problems  and  cost  overruns.  In  contrast,  the 
stakeholders  of  a  successful  DoD  program  made  a  firm  decision  to  modify  system 
requirements  and  not  commercial  items.  The  program  delivered  the  basic  capability  in  90 
days  for  20%  of  the  cost  of  a  previous  unsuccessful  effort  to  build  the  same  system.  The 
failure  of  the  previous  effort  was  attributed  to  extensive  modification  of  commercial  items. 

3.1.4  If  the  gap  is  too  great,  commercial  items  may  not  be  appropriate.  Some  attempts  to 
modify  DoD  requirements  to  facilitate  the  use  of  commercial  items  have  been  unsuccessful. 

In  one  case,  an  initial  decision  to  embrace  commercial  business  practices  was  abandoned 
because  the  DoD  organization  was  already  employing  more  mature  practices  than  those 
supported  by  commercial  items.  The  program  office  made  a  reasonable  decision  to  commit  to 
custom  development  and  ongoing  sustainment.  In  another  case,  a  commercial  item  was 
substantially  enhanced  to  address  unique  DoD  practices.  The  enhancements  were  so 
significant  that  essentially  a  custom  system  was  delivered.  The  vendor  justifiably  would  not 
support  the  item  under  the  standard  commercial  maintenance  agreement.  The  program  had  to 
contract  with  the  vendor  for  unique  support  services  for  the  life  of  the  system. 

3.1.5  Buy-in  from  key  stakeholders  is  critical.  It  is  not  enough  to  issue  a  top-down  directive 
that  the  organization  implement  a  new  business  practice.  Key  stakeholders  should  understand 
and  accept  the  extent  of  the  change  required.  This  means  that  stakeholders  must  be  involved 
very  early  in  the  process.  Some  programs  have  misjudged  the  importance  of  involving  all 
stakeholders.  In  one  organization,  the  DoD  end  users  agreed  to  change  their  business  practices 
and  adopt  those  made  necessary  by  use  of  a  commercial  item.  However,  one  key  group  of 
stakeholders  was  not  consulted.  This  group,  a  commercial  firm  that  provided  distribution 
services  under  a  different  commercial  model,  rejected  the  new  practices  and  forced  a  return  to 
business  as  usual.  Eventually  the  program  was  cancelled.  In  another  program,  end  users 
stopped  deployment  of  the  system  by  identifying  deficiencies  in  the  look  and  feel  of  a 
commercial  item  during  operational  test  and  evaluation.  In  contrast,  a  successful  program 
employed  external  experts  who  understood  the  ramifications  of  the  commercial  item  on 

"  Level  5  DII  COE  compliance  requires  segmented  applications,  installation  using  COE  tools,  and  operation 
with  the  COE  kernel.  Level  7  requires  that  applications  not  duplicate  functionality  provided  by  COE  services. 
See  [  C4I  STANDARDS]. 
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business  practices.  These  experts  set  clear  expectations  and  helped  stakeholders  make  the 
necessary  adjustments  to  use  the  system. 

3.1.6  Requirement  specifications  must  be  fiexible  and  negotiable.  A  traditional  development 
model  that  specifies  all  system  requirements  prior  to  considering  the  capabilities  available  in 
the  marketplace  is  ill  suited  to  the  development  of  systems  incorporating  commercial  items. 
The  resulting  requirements  are  unlikely  to  be  sufficient  for  the  selection  of  appropriate  items 
because  they  do  not  incorporate  the  broad  range  of  marketplace,  vendor,  and  product 
characteristics  that  must  be  considered  in  the  selection  process.  Many  significant 
characteristics  only  become  evident  during  market  research  and  evaluation  of  commercial 
items.  In  addition,  without  flexibility  in  requirements,  it  is  unlikely  that  any  commercial  item 
will  suit  program  needs.  One  program  demonstrated  little  willingness  to  leave  some  details 
undefined  until  later  in  the  development  proeess.  The  program  offiee  and  the  end  user 
attempted  to  finalize  all  system  requirements  in  advanee  of  market  research.  This  increased 
the  gap  between  the  eommercial  item  offerings  and  the  documented  requirements.  As  a  result, 
the  program  struggled  to  identify  and  ineorporate  commereial  items.  In  contrast,  a  successful 
program  pared  down  requirements  to  reflect  essential,  as  opposed  to  customary  or  preferred, 
business  practices.  This  allowed  for  flexibility  in  ehoosing  an  aeeeptable  commercial  item. 

3.1.7  New  approaches  to  program  management  can  enable  increased  use  of  commercial 
items.  Many  program  managers  find  that  to  maximize  the  use  of  eommercial  items,  they  must 
“invent”  a  unique  approach  to  requirements  management.  Within  a  month  of  embarking  on  its 
development  path,  the  integrated  product  team  (IPT)  of  a  successful  program  erafted  a  new 
development  process  that  sought  to  maximize  the  use  of  appropriate  commercial  items.  This 
process  identified  both  strengths  and  shortfalls  in  the  capabilities  provided  by  commercial 
items.  This  information  allowed  the  program  to  develop  strategies  to  mitigate  the  shortfalls, 
while  at  the  same  time  identifying  opportunities  for  improved  business  practices  within  the 
DoD  organization.  In  another  suecessful  program,  the  IPT  was  used  as  a  mechanism  for 
identifying  and  making  tradeoffs  among  system  context,  arehiteeture  and  design,  and  the 
capabilities  of  commercial  items.  Requirements  were  collected  and  prioritized,  and  costs  were 
estimated  based  on  the  commercial  availability  of  the  required  capability.  This  information 
was  used  to  rework  program  priorities.  A  third  successful  program  developed  an  unusual 
incentive  strategy  that  rewarded  individual  engineers  for  identifying  eommereial  items  that 
could  be  used  in  the  system. 

Suggestions 

The  following  suggestions  can  help  organizations  embrace  commercial  business  practices; 

— To  understand  the  marketplace 

•  Conduct  market  research  independent  of  the  contractor. 

•  Identify  all  significant  commercial  players  in  the  relevant  application  area. 

•  Participate  in  the  relevant  conferences,  trade  shows,  and  user,  professional,  and 
standards  groups. 

•  Identify  the  technology  domains  represented  by  the  application  area. 

— To  understand  the  system  context 

•  Track  changes  to  all  commercial  item  guidelines  and  direction  from  the  DoD. 

•  Reference  these  guidelines  and  direction  in  contract  specifications. 
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•  Propose  changes  to  guidelines  and  direction  to  reflect  new  commercial  items  needed 
in  the  system  context. 

•  Maintain  a  flexible  view  of  requirements  and  business  practices. 

•  Identify  all  of  the  stakeholders  and  involve  them  early. 

•  Pare  down  stated  requirements  to  reflect  only  essential  stakeholder  needs. 

— To  bridge  the  gap 

•  Determine  the  gap  between  the  capabilities  and  services  provided  in  the  marketplace 
and  those  required  by  the  system. 

•  Include  the  vendor  in  tradeoff  discussions  when  possible. 

•  Provide  incentives  to  encourage  the  contractor  to  investigate  all  solutions  that  lead  to 
the  appropriate  outcome. 

•  Don’t  modify  the  commercial  item. 

•  Plan  for  a  life-cycle  support  system  for  any  modified  commercial  item. 

•  Plan  to  make  repeated  tradeoffs  among  the  system  context,  the  architecture  and 
design,  and  the  capabilities  in  the  marketplace. 

•  Document  all  tradeoffs  made. 

•  Provide  early  functional  demonstrations  to  get  stakeholder  buy-in. 

3.2  Evaluating  commercial  items 

In  some  cases,  commercial  item  evaluation  is  performed  as  part  of  source  selection.  This  is  a 
highly  constrained  form  of  evaluation  that  must  be  conducted  only  in  accordance  with  source 
selection  criteria  and  the  source  selection  plan.  However,  the  definition  of  evaluation  applied 
in  this  document  is  far  broader.  Evaluation  is  also  necessary  to  assist  in  identifying 
commercial  capabilities  when  defining  source  selection  criteria,  in  choosing  among  alternate 
architectures  and  designs,  in  determining  whether  new  releases  continue  to  meet 
requirements,  and  in  ensuring  that  the  commercial  items  function  as  expected  when  linked  to 
other  system  components.  These  forms  of  commercial-item  evaluation  provide  critical 
information  about  the  tradeoffs  among  system  context,  architecture  and  design,  and 
commercial  capabilities.'^  Unfortunately,  evaluating  commercial  items  in  order  to  identify 
system  tradeoffs  is  an  unfamiliar  process  for  many  program  managers  (and  their  users).  It  is 
equally  imfamiliar  for  many  contractors  who  are  more  comfortable  with  simply  meeting  a 
specified  set  of  requirements. 

3.2.1  It  is  critically  important  to  evaluate  sU.  aspects  of  a  commercial  item.  After  a 
commercial  item  is  selected,  characteristics  of  the  item  and  the  vendor  become  integral  parts 
of  the  system.  Characteristics  such  as  security  and  information  assurance,  inter-operability, 
reliability,  and  maintainability  are  of  particular  importance.  One  program  selected  commercial 
items  with  the  expectation  that  the  vendor  would  provide  the  necessary  maintenance 
capabilities.  However,  the  vendor’s  commercial  support  strategy  did  not  provide  the  spares, 
training,  or  repair  cycles  necessary  for  military  use.  The  program  was  left  with  a  choice: 
redesign  the  system  or  buy  the  additional  capability.  Other  programs  struggled  because  they 
did  not  evaluate  concerns  such  as  the  vendor’s  financial  stability  and  strategic  direction,  the 


There  is  a  wide  range  of  techniques  that  can  be  used  to  evaluate  commercial  items  in  support  of  these 
tradeoffs.  See  [EVALUATION  TUTORIAL]  for  a  discussion  of  some  of  these  evaluation  techniques. 
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volatility  of  the  technology  on  which  the  commercial  item  was  based,  or  the  frequency  of 
commercial-item  releases.  On  the  other  hand,  successful  programs  deliberately  considered 
these  and  other  characteristics  of  the  commercial  item,  the  vendor,  and  the  marketplace  (e.g., 
positioning  and  innovation)  in  the  evaluation  process. 

3.2.2  Evaluating  various  commercial  items  means  comparing  things  that  may  not  compare 
very  well.  A  vendor’s  success  in  the  marketplace  depends  on  its  ability  to  offer  unique 
capabilities  to  attract  customers.  This  often  results  in  products  within  a  market  sector  that  are 
functionally,  architecturally,  or  technically  different.  One  program  evaluated  two  technologies 
in  order  to  determine  which  one  was  best  suited  for  use  in  a  communication  system.  The 
program  found  that  the  technologies  could  not  be  directly  compared  because  each  reflected 
different  assumptions  about  the  system  in  which  the  technology  would  be  used.  In  order  to 
conduct  the  evaluation,  the  program  first  had  to  define  system  architectures  that  reflected  the 
best  use  of  each  technology.  Then,  these  architectures  were  evaluated  against  the 
characteristics  desired  by  the  program.  A  decision  on  a  specific  commercial  implementation 
could  only  be  made  after  the  system  architecture  was  selected. 

3.2.3  Commercial  items  are  not  always  commercial  off-the-shelf  (COTS).  The  FAR 
definition  of  commercial  items  allows  each  program  great  latitude.  Yet,  the  desired  benefits 
from  selecting  commercial  items  are  maximized  when  the  items  fit  the  more  narrow  definition 
of  COTS.  Programs  have,  on  occasion,  purchased  commercial  items  they  assumed  to  be 
COTS  that  were  really  versions  of  systems  used  in-house  or  custom-produced  for  another 
organization.  In  one  case,  the  one-of-a-kind  item  purchased  did  not  represent  best  commercial 
practice  and  had  no  user  base  or  established  distribution  and  support  system.  The  program 
was  subsequently  cancelled.  In  another  case,  a  contractor  claimed  that  dozens  of  commercial 
items  were  being  incorporated  into  a  system;  the  program  wrongly  assumed  that  the 
commercial  items  were  COTS.  A  post-delivery  examination  exposed  these  items  to  be  little 
more  than  contractor-specific  tools  and  scripts.  As  a  result  of  these  contractor-specific  items, 
the  program  was  unable  to  reconstruct  the  system  without  the  long-term  support  of  the 
contractor — an  outcome  they  had  hoped  to  avoid. 

3.2.4  Even  unavoidable  tailoring  of  commercial  items  can  increase  program  risk.  Some 
commercial  items  are  designed  for  custom  tailoring.  For  example,  database  systems  often 
require  a  specialized  schema  tailored  to  the  user’s  data  and  applications.  Likewise,  passenger 
aircraft  are  designed  to  accommodate  unique  seating  arrangements,  engines,  and 
communications  packages.  Although  such  tailoring  is  a  necessary  and  acceptable  way  of 
doing  business,  any  modification,  including  tailoring,  can  be  a  maintenance  liability.  One 
program  purchased  a  commercial  item  that  required  extensive  tailoring  to  meet  system 
requirements.  Program  managers  thought  they  had  purchased  a  system  solution.  Rather,  they 
had  committed  to  development  and  maintenance  of  scripts  written  in  a  fourth-generation 
programming  language.  Significant  effort  was  required  to  rework  the  scripts  every  time  the 
commercial  item  was  updated.  In  addition,  the  program  was  forced  to  commit  to  long-term 
use  of  the  commercial  item  in  order  to  protect  its  investment  in  scripts.  Another  program 
found  that  development  of  tailored  scripts  was  complex  and  required  the  same  careful 
engineering  and  management  as  traditional  software-development  activities.  In  addition,  the 
ability  to  tailor  commercial  items  invited  ad  hoc  requirement  changes  and  customization  by 
end  users. 
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3.2.5  Incomplete  evaluation  of  commercial  items  will  affect  program  planning  in 
unexpected  ways.  Realizing  the  promise  of  reduced  cycle  times  and  lower  program  costs 
requires  detailed  insight  into  the  capabilities  of  commercial  items  and  the  fit  of  those  items 
within  the  context  of  the  integrated  system  and  the  program  schedule.  It  is  an  unfortunate 
commercial  practice  for  vendors  to  hype  new  items  or  versions  long  before  they  are  ready  for 
release  (this  is  one  reason  for  the  term  “vaporware”).  In  fact,  vendors  often  use  marketplace 
reaction  to  press  releases  about  upcoming  products  to  decide  whether  or  not  to  even  produce 
the  commercial  item.  One  DoD  project  based  its  system  schedules  on  the  very  optimistic 
promises  made  by  vendors.  The  project  was  behind  schedule  at  its  inception  because 
deliveries  promised  by  vendors  for  the  start  of  the  program  turned  out  to  be  many  months  late. 
Other  programs  fell  victim  to  vendor  claims  of  a  fully  capable  commercial  iteni  only  to  later 
find  that  many  months  of  vendor  effort  were  needed  to  prepare  the  item  for  delivery.  A 
detailed  evaluation  of  vendor  claims  and  a  careful  assessment  of  the  program  risks  inherent  in 
claims  that  could  not  be  verified  should  have  been  conducted  as  part  of  product  evaluation. 

3.2.6  Evaluation  will  be  repeated  many  times  during  the  life  of  the  system.  Over  the  life  of  a 
DoD  system,  new  versions  of  commercial  items  will  become  available  and  the  vendor’s 
business  practices  will  typically  change  rapidly.  In  addition,  some  vendors  may  withdraw 
their  commercial  items  from  the  marketplace,  while  other  vendors  may  enter  with  new 
items.'^  Each  of  these  changes  may  call  for  a  new  evaluation  to  determine  whether  the 
commercial  item  will  continue  to  meet  current  system  requirements  or  whether  a  new  item 
adds  desired  capabilities  or  performance.  One  commercial  organization  consistently  failed  to 
reevaluate  the  security  characteristics  of  new  releases  (versions)  that  provided  access  to  a 
sensitive  proprietary  database.  A  security  expert  later  expressed  the  concern  that  the 
corporation  had  not  experienced  any  serious  security  breaches  only  because  it  had  been  lucky. 
A  DoD  program  did  not  reconsider  a  decision  to  buy  a  commercial  item  when  plans  to  change 
end-user  business  practices  were  dropped.  As  a  result,  the  program  invested  sigmficant  effort 
in  modifying  the  commercial  item  to  bridge  a  newly  opened  gap.  On  the  other  hand,  another 
DoD  program  reconsidered  a  commercial  item  two  years  after  it  was  rejected  because  of  the 
vendor’s  pricing  strategy.  Over  the  period,  the  vendor’s  pricing  strategy  had  changed. 
Subsequently,  the  commercial  item  was  selected  and  incorporated  into  the  system. 

3.2.7  A  test  bed  is  an  excellent  mechanism  for  gaining  insight  into  the  design  and  behavior 
of  a  commercial  item.  Vendors  do  not  typically  provide  detailed  information  about  such  areas 
as  commercial-item  architecture,  design,  implementation,  performance,  and  limitations.  This 
lack  of  visibility  into  the  workings  of  commercial  items  can  hamper  efforts  to  evaluate  them 
for  use  within  a  larger  system.  One  program  found  that  building  a  test  bed  was  a  necessary 
step  in  product  evaluation.  Unless  it  was  impractical  to  do  so,  the  program  did  not  buy  any 
commercial  item  that  had  not  been  evaluated  in  this  test  bed.  Another  program  found  that 
participation  in  test  beds  by  the  end  users  allowed  those  users  to  contribute  to  decisions 
regarding  tradeoffs  among  commercial  item  function,  cost,  and  other  factors.  Other  programs 
found  that  cost  and  schedule  benefits  accrued  when  test  beds  were  used  to  discover  program 
risks  before  significant  rework  was  necessary.  However,  one  program  had  unrealistic 
expectations  about  the  cost  and  schedule  savings  that  would  result  from  the  use  of  test  beds. 
The  program  deployed  the  test  bed  directly  into  a  war  zone  as  an  early  demonstration  of  the 


Commercial  software  vendors  generally  have  a  release  cycle  of  18  months  or  less  in  order  to  stay  competitive 
in  the  marketplace. 
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technology.  The  program  expected  that  money  spent  on  the  war  zone  test  bed  would  result  in 
decreased  funding  requirements  during  full-scale  development  of  the  system.  Instead,  system 
costs  went  up  and  the  schedule  was  extended  as  end  users  drove  operational  capability  in  a 
different  direction  than  was  planned  for  the  program. 

Suggestions 

The  following  suggestions  can  be  helpful  for  evaluating  commercial  items: 

— To  develop  the  skills  needed 

•  Employ  outside  experts  to  support  program-office  evaluation  activities. 

•  Train  the  program  office  and  the  stakeholders  on  how  to  evaluate  commercial  items. 

•  Repeat  this  training  as  personnel  or  the  nature  of  the  commercial  items  being 
evaluated  change. 

•  Select  a  contractor  who  has  past  experience  in  evaluating  commercial  items. 

— To  conduct  evaluations 

•  Decide  in  advance  what  information  you  want  to  gain  from  the  evaluation  of  a 
commercial  item. 

•  Select  evaluation  techniques  based  on  the  type  of  information  required  and  the 
importance  of  the  selection  to  the  program. 

•  Unless  it  is  impractical,  evaluate  potential  commercial  items  in  a  system  test  bed. 

•  Consider  both  the  capabilities  of  the  commercial  item  and  the  business  practices  of  the 
vendor. 

•  Take  into  account  the  business  motivations  of  the  vendors. 

•  Understand  the  vendor’s  strategy,  and  talk  to  other  buyers. 

•  Understand  where  you  stand  in  relation  to  the  vendor’s  other  customers. 

•  Budget  for  repeated  evaluations  throughout  the  program’s  life  cycle. 

3.3  Working  with  Contractors  and  Vendors 

Programs  are  most  effective  in  working  with  vendors  when  a  program  adopts  practices  and 
expectations  that  are  familiar  to  vendors.  Further,  not  only  must  the  program  act  like  a 
commercial  organization,  but  it  must  also  expect  to  be  treated  like  a  commercial  organization 
by  the  vendor.  Some  program  managers  have  expressed  fhistration  that  vendors  do  not  react 
to  program  needs  and  direction.  Other  program  managers  have  tried  to  use  the  same 
techniques  with  vendors  that  had  been  successful  when  applied  to  contractors  and 
subcontractors — ^usually  with  disappointing  results.  It  is  incumbent  on  the  program  manager 
to  determine  how  important  the  program  is  to  a  specific  vendor  as  part  of  the  commercial-item 
evaluation.  The  program  manager  can  use  this  knowledge  to  establish  an  appropriate 
relationship  with  the  vendor.  In  some  cases,  the  program  manager  can  influence  the  vendor  to 
be  responsive  to  unique  program  needs  (e.g.,  by  incorporating  new  features  into  the 
commercial  item).  At  the  same  time,  the  DoD’s  unique  requirements  and  expectations  will  not 
always  sway  the  vendor.  In  this  case,  the  program  manager  should  revisit  requirements  and 
expectations  to  make  sure  they  are  absolutely  necessary  and,  where  appropriate,  work  to 
adjust  them  to  allow  the  use  of  commercial  items*"^. 


The  DoD  must  justify  the  need  for  a  particular  requirement,  including  an  analysis  of  alternate  means  for 
satisfying  a  requirement,  before  commencing  an  acquisition.  However,  it  is  difficult  to  perform  a  good  analysis 
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3.3.1  Tailor  contractual  relationships  to  the  realities  of  the  marketplace.  The  use  of 
commercial  items  means  that  the  program  office  and  the  contractor  will  need  new  skills. 
Contractors  have  traditionally  been  selected  for  their  ability  to  build  custom  systems,  not  for 
their  knowledge  of  the  marketplace,  expertise  with  specific  commercial  items,  or  ability  to 
integrate  items.  A  contractor’s  ability  in  these  new  areas  and  its  ability  to  rapidly 
accommodate  frequent  technology  changes  will  likely  be  as  significant  as  the  traditional 
factors  considered  in  source  selection.  In  addition,  the  contractor’s  business  relationships  and 
vendor  alliances  will  now  have  the  same  significance  as  a  large  cadre  of  skilled  labor.  Also,  a 
vendor  is  most  concerned  with  meeting  the  needs  of  the  wider  marketplace,  even  when  a  DoD 
program  represents  that  vendor’s  largest  single  customer.  One  procuring  organization  was 
strongly  motivated  to  apply  innovative  commercial  buying  practices  for  purchasing  a 
commercial  item.  However,  the  organization  expected  to  fit  those  commercial  buying 
practices  within  the  framework  of  the  traditional  government  procurement  process  (e.g., 
government  specifications,  paperwork,  test  programs,  etc.).  The  organization  assumed  that  the 
vendor  would  adapt  to  the  government  bureaucracy  when  in  fact  it  was  necessary  for  the 
program  to  adapt  to  the  buying  practices  used  in  the  commercial  marketplace.  The  program 
found  that  every  time  the  vendor  adapted  to  suit  the  government,  the  cost  of  the  commercial 
item  increased. 

3.3.2  Program  decisions  should  reflect  total  ownership  cost.^^  Both  commercial  and  DoD 
programs  frequently  underestimate  the  unique  sustainment  costs  associated  with  commercial 
items.  These  costs  include  market  research,  evaluation,  test  and  integration  for  version 
upgrade,  commercial-item  replacement,  technology  refresh,  and  annual  licensing  fees.  An 
unsuccessful  program  failed  to  capture  the  sustainment  costs  for  commercial  items. 
Sustainment  costs  were  the  responsibility  of  a  different  part  of  the  organization.  This  part  of 
the  organization  was  consistently  under-funded,  and  upgrades  to  deployed  systems  were 
delayed  because  of  a  lack  of  resources  for  testing  and  integration  of  new  commercial  releases. 
Another  program  was  forced  to  slip  a  very  aggressive  schedule  because  training, 
documentation,  purchase,  and  installation  costs  were  not  considered  in  making  budget 
decisions.  Successful  DoD  programs  address  many  of  the  issues  regarding  cost  by  embracing 
total  ownership  cost  models  that  incorporate  both  routine  sustainment  costs  and  costs  for 
frequent  technology  updates  of  commercial  items. 

3.3.3  Vendors’  price  models  are  incompatible  with  familiar  DoD  cost  models.  The  DoD 
traditionally  uses  cost  models  that  are  constructed  from  labor  hours  and  materials  plus  profit. 
However,  the  price  of  a  commercial  item  is  determined  by  other  marketplace  factors.  New 
price-based  techniques  are  necessary.  Program  managers  often  have  little  experience  in 
determining  whether  quoted  prices  are  reasonable.  One  program  was  imable  to  streamline  the 
procurement  process  nearly  as  much  as  anticipated  because  of  a  lack  of  experience  in 
considering  marketplace  factors  when  conducting  price-based  analysis.  Another  program 
found  that  government  financial  managers  could  not  determine  the  reasonable  price  for 
modifications  to  a  commercial  item — ^no  comparisons  with  similar  contracts  or  price  history 
information  were  available.  On  the  other  hand,  most  programs  discovered  a  much  broader 


of  alternate  means  without  a  thorough  evaluation  of  commercial  items  in  the  marketplace.  In  practice,  the 
maimer  in  which  requirements  are  stated  can  remove  commercial  items  from  consideration. 

For  a  definition  of  total  ownership  cost,  see  [TOC]. 
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selection  of  relatively  low-cost  commercial  offerings  when  traditional  cost  accounting  data 
was  no  longer  required. 

3.3.4  Key  vendors  can  be  strategic  partners.  Do  not  assume  that  the  contractor  alone  has 
sufficient  insight  into  the  commercial  items.  Where  possible,  it  is  important  to  involve 
vendors  directly.  Relationships  with  key  vendors  can  take  many  forms.  After  a  failed  attempt 
at  building  a  system,  one  program  office  determined  that  extensive  custom  modifications  to  a 
commercial  item  were  largely  to  blame.  For  a  subsequent  attempt,  a  strict  rule  was  adopted  to 
resist  any  modifications  to  the  selected  commercial  items.  The  program  office  requested  that 
the  vendor  not  make  any  DoD-specific  modifications.  The  program  made  suggestions  to  the 
vendor,  but  asked  that  the  vendor  only  make  a  change  if  it  made  sense  commercially.  The 
vendor  checked  on  the  viability  of  any  enhancements  by  asking  other  buyers.  This  successful 
program  was  able  to  influence,  rather  than  direct  the  vendor.  Other  programs  found  that 
including  vendors  as  part  of  integrated  product  teams  helped  foster  a  more  trusting  partnership 
among  the  vendor,  contractor,  and  program  office. 

3.3.5  Licenses  and  data  rights  define  the  relationship  with  the  vendor.  Licensing  is  the 
primary  vehicle  for  securing  the  use  of  commercial  items  such  as  software;  data  rights  are 
marketplace  vehicles  for  protecting  a  vendor’s  intellectual  property.’®  License  agreements  and 
data  rights  can  and  should  be  negotiated.  One  program  expressed  frustration  that  the  de  facto 
selection  of  a  commercial  item  had  already  been  made  prior  to  release  of  the  solicitation 
because  of  the  beneficial  pricing  arrangements  from  previously  negotiated  enterprise  licenses. 
While  the  larger  organization  saved  money  in  negotiating  one  set  of  licenses  covering  use  by 
many  programs,  this  practice  limited  the  individual  program’s  flexibility  in  choosing  the  most 
appropriate  commercial  item  for  the  system.  Another  program  neglected  to  negotiate  for  all 
necessaiy  licenses  as  part  of  the  initial  procurement.  After  the  commercial  item  was  selected 
and  system  development  began,  the  vendor’s  price  for  additional  licenses  increased 
dramatically. 

3.3.6  Commercial  items  frequently  come  with  little  technical  data.  Vendors  rarely  produce 
technical  data  in  a  format  that  can  be  used  by  a  program — much  of  this  data  exists  only  in  the 
minds  of  the  engineers  who  developed  the  item.  Therefore,  it  is  difficult  for  the  vendor  to 
share  the  technical  data  that  is  necessary  for  thorough  evaluation  or  that  is  traditionally 
required  to  operate  and  maintain  DoD  systems.  Even  when  technical  data  exists  in  formats 
consistent  with  program  expectations,  the  cost  can  be  prohibitive  because  vendors  are 
protective  of  their  intellectual  property.  Yet  some  commercial  items  are  so  critical  to  the 
system  that  the  program  must  be  protected  from  a  vendor’s  potential  unwillingness  or 
inability  to  support  older  releases  of  the  product  through  the  life  of  the  system.  Some 
programs  found  that  an  agreement  to  put  technical  data  in  an  escrow  account  (rather  than 
purchasing  technical  data  directly)  was  a  cost-effective  compromise.  However,  one  program 
never  checked  that  the  escrow  account  was  set  up  and  maintained  by  the  vendor.  When  the 
vendor  went  out  of  business,  the  program  was  forced  to  gather  what  technical  data  it  could 
from  personnel  who  had  previously  worked  for  the  vendor.  On  the  other  hand,  successful 
programs  negotiated  terms  of  the  escrow  to  include  the  essential  data  and  contingencies. 


With  a  license,  only  rights  to  use  the  current  version  are  conveyed.  This  is  not  the  same  as  “buying”  the 
commercial  item. 
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audited  the  escrow  account  regularly  to  make  sure  the  data  was  current  and  complete,  and 
budgeted  for  the  cost  of  the  escrow  throughout  the  life  of  the  system. 

3.3. 7  Programs  frequently  overestimate  the  impact  they  can  have  on  vendors.  The 
relationship  between  the  program  and  the  vendor  is,  in  most  instances,  very  different  from  the 
relationship  with  a  contractor.  While  contract  incentives  shape  the  relationship  with  a 
contractor,  the  vendor  is  selling  a  product,  seldom  program-unique  services.  Yet,  programs 
have  been  successful  in  influencing  product  changes.  In  one  case,  a  program  worked  as  part  of 
a  users  group  to  influence  other  customers  to  support  changes  needed  by  the  DoD  and  the 
vendor  implemented  the  widely  supported  changes.  On  the  other  hand,  other  vendors  agreed 
to  customize  software  in  anticipation  of  a  huge  number  of  DoD  licenses.  Two  factors  soured 
the  relationship:  changes  were  more  widespread  than  planned  and  the  number  of  licenses  the 
DoD  purchased  was  a  small  fraction  of  what  the  vendors  originally  anticipated.  The  DoD 
contract  was  thus  less  important  to  the  vendors,  which  program  managers  believed  resulted  in 
reduced  capability  and  poor  quality  in  the  customized  commercial  item.  Other  programs  were 
convinced  that  changes  to  commercial  items  would  be  included  in  subsequent  commercial 
releases.  In  several  cases  the  custom  enhancements  never  became  part  of  the  commercial 
item;  the  programs  had  to  choose  whether  to  maintain  a  unique  version  of  the  commercial 
item,  or  redesign  the  system  without  the  modifications. 

3.3.8  Consider  long-term  sustainment  before  modifying  commercial  items.  Custom 
modifications  to  a  commercial  item,  even  if  implemented  by  the  vendor,  result  in  custom 
items^’.  In  the  absence  of  specific  contractual  agreements,  the  vendor  has  little  incentive  to 
maintain  custom  enhancements.  One  organization  found  that,  even  with  a  maintenance 
contract,  updates  to  a  custom  version  lagged  significantly  behind  the  vendor’s  commercial 
releases  and  users  were  forced  to  live  with  older  (customized)  versions  of  the  commercial 
item*^.  In  another  case,  the  vendor  was  not  willing  to  maintain  the  unique  version  and  the 
program  office  was  incapable  of  maintaining  it.  Thus,  no  one  was  both  willing  and  able  to 
shoulder  the  burden  for  long-term  maintenance.  A  successful  organization  that  built  a  number 
of  systems  based  on  commercial  items  refused  any  modification  of  commercial  items  as  not 
maintainable  at  reasonable  cost. 

Suggestions 

The  following  suggestions  can  help  organizations  implement  commercial  buying  practices: 

— To  adjust  buying  practices 

•  Train  financial  management  and  contract  personnel  in  commercial  buying  practices. 

•  Adapt  business  and  engineering  models  and  acquisition  strategies  to  accommodate  the 
impact  of  using  commercial  items. 

— To  develop  and  execute  program  budgets 

•  Base  planning  on  total  ownership  cost  rather  than  catalog  price. 

•  Investigate  emerging  price  and  cost  models. 

”  The  definition  of  commercial  item  from  the  FAR,  Part  2,  allows  for  “minor”  modifications  made  to  meet 
federal  government  requirements.  In  light  of  problems  experienced  by  a  large  number  of  programs  that  have 
modified  commercial  items,  a  strong  position  against  modification  is  taken  here. 

**  For  a  discussion  of  whether  the  government  can  choose  to  ignore  upgrade  releases  of  a  commercial  item,  see 

section  3.4.3.  ,  j  n 

One  such  model  is  Constructive  COTS  (COCOTS),  a  cost  model  designed  by  Barry  Boehm  and  colleagues  to 

capture  the  most  important  costs  associated  with  COTS  component  integration.  See  [COCOTS] . 
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•  Perform  market  research  to  support  determinations  of  reasonable  value. 

•  Include  a  budget  and  schedule  for  unexpected  commercial  impact. 

— To  strengthen  program,  contractor,  and  vendor  relationships 

•  Verify  the  claims  made  for  commercial  items  by  vendors  and  contractors. 

•  Verify  the  availability  of  commercial  items. 

•  Examine  any  acquisition  strategy  to  see  where  it  can  be  made  more  flexible  or  better 
suited  to  the  unique  commercial  aspects  of  the  system  in  question. 

•  Use  contract  incentives  to  encourage  appropriate  relationships. 

•  Maintain  close  relationships  with  vendors  to  exploit  improvements  and  avoid 
surprises. 

3.4  Engineering  for  life-cycle  support 


The  importance  of  system  engineering  in  systems  that  integrate  a  number  of  commercial  items 
has  been  frequently  overlooked  or  underestimated.  At  least  one  program  struggled  because  it 
viewed  the  system  under  construction  as  simply  a  procurement  of  a  set  of  qualified  items. 
Systems  that  integrate  multiple  commercial  items  require  extensive  engineering  to  define 
system  architecture  with  a  modular  design  that  is  open  enough  to  facilitate  the  insertion  of 
new  commercial  technology.  This  is  not  a  “one  time”  activity  because  changes  in  commercial 
items  and  in  the  marketplace  may  drive  frequent  reengineering  of  the  system  throughout  the 
life  of  the  program.  The  program  manager  must  expect  to  analyze  requirements,  evaluate 
commercial  items,  and  design,  integrate,  and  test  the  system  at  various  points  in  the  life  of  the 
system.  Failure  to  evolve  the  architecture  and  reengineer  the  system  to  address  changes  in 
commercial  items  and  the  marketplace  will  potentially  result  in  a  system  that  cannot  be 
maintained  as  vendors  drop  support  for  obsolete  commercial  items. 

3.4.1  Commercial  items  can  drive  the  system  architecture  and  design.  Frequent  changes  in 
commercial  items  and  their  underlying  technologies  create  a  particular  challenge  for  defining 
a  system  architecture.  The  architecture  must  be  flexible  enough  to  incorporate  new  releases  of 
commercial  items  and  to  remove  obsolete  commercial  items  as  necessary.  One  otherwise 
successful  program  selected  a  system  architecture  that  was  dependent  on  a  specific 
commercial  item.  Unfortunately,  the  vendor  went  out  of  business.  The  program  evaluated 
both  commercial  and  noncommercial  options  to  replace  the  obsolete  item,  but  there  were  no 
acceptable  alternative  items.  It  was  discovered  that  any  change  in  the  obsolete  item  would 
invalidate  many  other  design  decisions.  The  program  was  forced  to  retain  the  obsolete 
commercial  item  in  spite  of  the  new  maintenance  burden.  By  carefully  selecting  the  system 
architecture,  another  program  was  able  to  replace  various  commercial  items  in  a  system 
through  a  succession  of  major  updates.  The  updates  included  replacing  the  software  language, 
the  computer  hardware,  and  the  major  communications  protocols. 

3.4.2  Integrating  commercial  items  requires  extensive  expertise.  Although  the  expertise  is 
growing,  relatively  few  programs  or  contractors  have  extensive  experience  integrating 
commercial  items  into  DoD  systems.  Knowledge  of  both  the  system  context  and  each  selected 
commercial  item  is  necessary.  One  program  assumed  that  heterogeneous  commercial  items 
could  be  integrated  with  relatively  minimal  effort.  The  program  neglected  the  hard 
engineering  work  needed  to  develop  realistic  integration  and  test  schedules,  to  specify 
acceptance  criteria  for  the  system,  or  to  plan  for  long-term  system  evolution.  These  oversights 
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resulted  in  unhappy  users,  finger-pointing  between  the  vendors  and  the  program  office,  and 
cost  and  schedule  overruns.  Another  program  found  that  predicting  the  performance  of  a 
system  composed  of  a  number  of  integrated  commercial  and  custom  items  was  difficult. 
Prototyping  and  incremental  testing,  which  might  have  identified  performance  problems,  was 
not  used;  performance  problems  were  not  uncovered  until  operational  test  and  evaluation.  As 
a  result,  modifications  to  the  system  were  required  and  the  number  and  duration  of  test 
missions  more  than  tripled.  Several  other  programs  found  that  unique  technical  expertise  was 
required  to  integrate  commercial  items  because  the  internal  architectural  and  usage 
assumptions  of  the  items  were  unknown. 

3.4.3  Plan  for  obsolescence  and  upgrades.  It  is  tempting  to  assume  that  a  program  office  can 
avoid  the  problems  associated  with  upgrade  by  simply  continuing  to  deploy  an  older  release 
of  a  commercial  item.  While  this  may  be  true  for  some  hardware  items,  it  is  rarely  the  case  for 
software  items,  where  new  and  desirable  capabilities  and  performance  are  frequently  added, 
bugs  are  fixed,  and  vendors  drop  maintenance  for  older  releases.  A  release  or  two  can 
sometimes  be  safely  skipped,  but  most  software  commercial  items  (and  many  hardware 
commercial  items)  must  eventually  be  upgraded,  if  only  because  of  dependencies  on  other 
system  components  that  must  be  upgraded.  Except  in  very  specific  cases,  the  DoD  is  normally 
ill  prepared  to  implement  the  necessary  changes  to  old  versions  of  commercial  items  in  order 
to  avoid  technical  obsolescence  and  keep  them  functioning — even  when  good  technical  data 
is  available.  However,  by  adopting  an  open  system  architecture^®  with  modular  designs, 
maintaining  close  relationships  with  commercial  item  vendors,  and  monitoring  the 
marketplace,  one  particularly  successful  program  not  only  avoided  technological 
obsolescence,  but  also  developed  a  “sparing”  model  that  reduced  the  cost  of  spare 
components  by  40%.  Several  programs  were  successful  by  deliberately  pre-planning  for 
frequent  upgrades  of  commercial  items,  technology  insertion,  and  retirement  of  obsolete 
items.  Of  course,  even  the  most  careful  planning  cannot  anticipate  all  exigencies,  such  as  a 
vendor  going  out  of  business  or  being  taken  over  by  a  larger  firm  with  different  priorities.  In 
the  words  of  one  program  manager,  you  have  to  “pick  a  horse  and  ride  it  until  the  legs  fall 
off”  But  you  should  also  be  ready  to  switch  horses. 

3.4.4  New  configuration  management  techniques  are  critical.  Frequent  changes  to 
commercial  items  have  caused  many  programs  to  maintain  multiple  configuration  baselines 
both  during  development  and  in  the  field.  This  places  imusual  demands  on  traditional 
configuration-management  processes  that  strive  to  maintain  a  single  configuration  baseline. 
Several  programs  that  depended  on  multiple  commercial  items  found  that  some  items  required 
specific  versions  of  other  items  in  order  to  interface  effectively.  Upgrading  one  commercial 
item  caused  a  chain  reaction  that  demanded  changes  to  other  commercial  items  within  the 
system.  Given  that  vendors  release  items  according  to  their  own  schedules,  the  programs 
needed  a  configuration-management  system  that  could  select  fi-om  among  multiple  versions  of 
commercial  items  in  order  to  construct  different  system  configurations.  One  program  that  was 
distributed  over  multiple  locations  attempted  to  maintain  only  one  version  of  the  system 
deployed  at  a  time.  This  was  impractical,  however,  because  the  process  of  updating  all  sites 
took  up  to  one  year.  Many  programs  foimd  that  individual  sites  were  not  always  willing  to 
upgrade  to  the  ktest  version  of  the  system.  There  were  many  valid  reasons  for  preferring 


The  Software  Engineering  Institute  has  done  considerable  investigation  into  this  topic.  See 
[OPENSYSTEMS]. 
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alternate  configurations  (e.g.,  compatibility  with  hardware  and  other  systems,  cost).  However, 
without  careful  management,  this  can  create  a  maintenance  nightmare  when  a  change  has  to 
be  propagated  across  all  configurations  (as  was  the  case  in  Y2K  updates). 

3.4.5  End-user  support  requires  careful  consideration.  Effective  implementation  of  large, 
complex  systems  requires  substantial  end-user  support  to  ensure  that  the  system  is 
implemented  correctly,  that  users  are  knowledgeable  in  using  the  system,  and  that  they  have 
access  to  customer  support  that  responds  to  their  questions  and  maintains  the  system.  This  is 
compounded  when  the  system  is  deployed  to  multiple  locations.  One  program  provided  user 
guidance  rather  than  enforce  a  strict  deployment  and  usage  policy  because  of  legitimate 
differences  between  deployment  sites.  Given  that  the  different  user  sites  operated  in  different 
ways,  it  was  important  for  the  sites  to  be  able  to  tailor  commercial  items  and  the  system  to 
their  needs  and  even  deploy  the  system  in  a  site-specific  manner.  The  program  office  acted  as 
a  center  of  expertise  on  how  the  sites  could  deploy  the  system,  and  as  a  clearinghouse,  letting 
the  sites  know  what  had  (and  had  not)  worked  at  other  sites.  In  fact,  the  program  delayed 
widespread  deployment  of  commercial  item  upgrades  until  the  new  version  was  evaluated  at 
selected  end-user  sites.  In  another  program,  the  end  users  found  that  the  training,  guidance, 
and  help-desk  support  provided  by  the  program  office  were  not  adequate  to  allow  the  end 
users  to  integrate  the  system  into  their  site-unique  environments.  Each  site  wrote  a  separate 
contract  with  the  vendor  to  tailor  the  support  provided.  This  was  effective  from  the  program- 
office  perspective  because  individual  sites  bore  the  cost  for  this  support.  However,  the 
objective  of  standardizing  the  DoD  business  practice  was  not  achieved  as  each  site  purchased 
separate  training  and  support  service,  and  the  total  ownership  cost  was  significantly  increased. 

3.4.6  Extensive  program  testing  of  commercial  items  may  be  required.  Programs  often 
underestimate  the  impact  of  testing  commercial  items.  Often  DoD  application  of  commercial 
items  requires  qualification  and  operational  testing  and  evaluation  (e.g.,  live-fire  testing)  to 
show  that  the  items  continue  to  perform  as  expected  in  unique  military  environments.  In 
addition,  if  the  commercial  item  has  been  modified,  regression  testing  at  the  system  level  may 
be  needed  to  ensure  that  the  modification  does  not  change  the  expected  performance  of  the 
system.  For  example,  some  programs  found  that  higher  performance  engines  could 
outperform  the  airframe,  while  others  found  that  faster  hardware  or  software  components 
could  introduce  timing  problems  or  security  holes.  Lack  of  insight  into  the  internal  workings 
of  the  commercial  item  changes  the  nature  of  the  test  program.  One  program’s  ability  to 
conduct  operational  test  and  evaluation  was  complicated  by  the  fact  that  data  normally 
generated  during  the  development  testing  was  not  available  for  analysis  by  the  operational  test 
team.  Another  program  that  was  using  multiple  commercial  items  found  that  even  basic, 
advertised  capabilities  of  commercial  items  had  to  be  tested  before  the  program  could  begin 
its  planned  integration  testing.  The  program’s  initial  plans  and  schedules  for  testing 
commercial  items  underestimated  the  effort  required  by  a  factor  of  six. 

Suggestions 

The  following  suggestions  can  help  organizations  engineer  their  systems  for  life-cycle 
support: 

— To  develop  the  needed  skills 

•  Train  the  system  engineers  on  the  challenge  of  integrating  commercial  systems. 
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•  Select  a  contractor  who  has  expertise  in  the  system  context,  the  commercial 
marketplace,  and  integration  of  independently  developed  components. 

•  Use  independent  contractors  who  are  familiar  with  the  marketplace  to  evaluate  the 
architecture  against  projected  technology  and  product  enhancements. 

— To  engineer  the  system  architecture 

•  Recognize  the  impact  that  available  commercial  items  will  have  on  the  system 
architecture. 

•  Review  the  architectural  alternatives  being  considered  and  the  factors  that  will  be  used 
to  select  the  system  architecture. 

•  Validate  assertions  made  about  the  flexibility  of  the  system  architecture. 

•  Plan  for  commercial-item  updates  and  technology  insertion  as  part  of  the  development 
cycle. 

•  Anticipate  periodic  reanalysis  and  redesign  of  the  system  and  evolution  of  the  system 
architecture. 

•  Align  contract  incentives  with  program  objectives  through  the  life  cycle. 

— To  manage  change 

•  Ensure  that  rigorous  configuration  management  is  exercised. 

•  Base  interfaces  on  publicly  recognized  industrial  standards  that  are  widely  supported 
in  the  marketplace. 

•  Monitor  the  marketplace  for  technology  advancements. 

•  Establish  plans  to  work  with  vendors  for  problem  resolution. 

— To  test  commercial  items 

•  Unless  it  is  impractical,  evaluate  potential  commercial  items  in  a  system  test  bed. 

•  Focus  test  beds  on  high-risk  items. 

•  Test  for  unanticipated  side  effects  in  areas  such  as  security,  safety,  reliability  and 
performance  from  commercial-item  upgrades. 
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4.  Summary 


There  are  no  “silver  bullets”  when  dealing  with  commercial  items.  While  there  are  significant 
benefits,  these  benefits  can  be  attained  only  by  understanding  and  addressing  the  significant 
new  challenges  that  are  driven  by  the  fundamental  differences  between  building  items  and 
buying  them.  It  must  also  be  emphasized  that  the  risks  associated  with  traditional  system 
development  do  not  disappear  simply  because  the  system  makes  use  of  commercial  items. 
This  last  point  is  critical:  no  matter  how  much  of  a  system  is  provided  by  commercial  items, 
the  overall  system  still  must  be  engineered,  developed,  integrated,  tested,  delivered,  sustained, 
and  managed. 

The  program  manager  is  ultimately  responsible  for  ensuring  that  the  system  meets  cost, 
schedule,  and  performance  parameters.  Success  in  using  commercial  items  to  meet  those 
parameters  requires  the  following:  being  an  informed  consumer,  planning  for  continuous 
evolution  of  your  system,  and  being  flexible  and  willing  to  negotiate  throughout  the  life  cycle 
of  the  system. 
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Appendix  A 


FAR  Definition  of  Commercial  Item: 

(a)  Any  item,  other  than  real  property,  that  is  of  a  type  customarily  used  for  nongovernmental  purposes 
and  that  — 

(1)  Has  been  sold,  leased,  or  licensed  to  the  general  public;  or, 

(2)  Has  been  offered  for  sale,  lease,  or  license  to  the  general  public; 

(b)  Any  item  that  evolved  from  an  item  described  in  paragraph  (a)  of  this  definition  through  advances 
in  technology  or  performance  and  that  is  not  yet  available  in  the  commercial  marketplace,  but  will 
be  available  in  the  commercial  marketplace  in  time  to  satisfy  the  delivery  requirements  under  a 
Government  solicitation; 

(c)  Any  item  that  would  satisfy  a  criterion  expressed  in  paragraphs  (a)  or  (b)  of  this  definition,  but  for 

(1)  Modifications  of  a  type  customarily  available  in  the  commercial  marketplace;  or 

(2)  Minor  modifications  of  a  type  not  customarily  available  in  the  commercial  marketplace 
made  to  meet  Federal  Government  requirements.  Minor  modifications  means 
modifications  that  do  not  significantly  alter  the  nongovernmental  function  or  essential 
physical  characteristics  of  an  item  or  component,  or  change  the  purpose  of  a  process. 
Factors  to  be  considered  in  determining  whether  a  modification  is  minor  include  the  value 
and  size  of  the  modification  and  the  comparative  value  and  size  of  the  final  product. 

Dollar  values  and  percentages  may  be  used  as  guideposts,  but  are  not  conclusive  evidence 
that  a  modification  is  minor; 

(d)  Any  combination  of  items  meeting  the  requirements  of  paragraphs  (a),  (b),  (c),  or  (e)  of  this 
definition  that  are  of  a  type  customarily  combined  and  sold  in  combination  to  the  general  public; 

(e)  Installation  services,  maintenance  services,  repair  services,  training  services,  and  other  services  if 
such  services  are  procured  for  support  of  an  item  referred  to  in  paragraphs  (a),  (b),  (c),  or  (d)  of  this 
definition,  and  if  the  source  of  such  services  — 

(1)  Offers  such  services  to  the  general  public  and  the  Federal  Government  contemporaneously 
and  under  similar  terms  and  conditions;  and 

(2)  Offers  to  use  the  same  work  force  for  providing  the  Federal  Government  with  such 
services  as  the  source  uses  for  providing  such  services  to  the  general  public; 

(f)  Services  of  a  type  offered  and  sold  competitively  in  substantial  quantities  in  the  commercial 
marketplace  based  on  established  catalog  or  market  prices  for  specific  tasks  performed  imder 
standard  commercial  terms  and  conditions.  This  does  not  include  services  that  are  sold  based  on 
hourly  rates  without  an  established  catalog  or  market  price  for  a  specific  service  performed; 

(g)  Any  item,  combination  of  items,  or  service  referred  to  in  paragraphs  (a)  through  (f), 
notwithstanding  the  fact  that  the  item,  combination  of  items,  or  service  is  transferred  between  or 
among  separate  divisions,  subsidiaries,  or  affiliates  of  a  contractor;  or 

(h)  A  nondevelopmental  item,  if  the  procuring  agency  determines  the  item  was  developed  exclusively 
at  private  expense  and  sold  in  substantial  quantities,  on  a  competitive  basis,  to  multiple  State  and 
local  governments. 
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